The present invention relates to image combining and encoding methods and devices, as well as imaging systems including these technologies, and more particularly to technologies to sequentially encode a composite image generated by combining a previous image and a current image.
In recent years, technologies for efficiently encoding moving pictures such as MPEG2/4 and H.264 have been actively studied and applied to diverse fields such as computers, communication, consumer audio/video (A/V) devices, and broadcasting. Meanwhile, as high-definition flat panel displays, including plasma displays and liquid crystal display televisions, are rapidly penetrating the market, moving pictures having high-definition television (HDTV) formats have been rapidly becoming popular, and thus the amount of data handled in a process of encoding moving pictures has been becoming very large. Particularly, in a field of consumer small cameras, such as movie cameras and digital still cameras which operate with small batteries, high-performance technologies for encoding moving pictures which can process data in a compact size and with a low power consumption have been actively studied.
Meanwhile, in order to introduce the pleasure of recording moving pictures more widely in the market, product lines of consumer small cameras such as movie cameras and digital still cameras include products which provide a distinguishing feature not only to record imaged images, but also to sequentially combine previous and current images while recording moving pictures so that trajectories of a moving object are drawn when taking moving pictures of a golf swing, a baseball bat swing, etc.
A general method to record moving pictures in such a way by combining previous and current images, and then by sequentially encoding moving pictures of the generated composite images will be described below.
FIG. 13 is a diagram schematically illustrating a typical image combining and encoding device which combines previous and current images, and sequentially encodes moving pictures of the generated composite images. The image combining and encoding device 101 includes an input terminal 2, a storage section 3, a configuration terminal 4, an image combining section 5, an encoding section 6, and an output terminal 7.
A current image is sequentially input to the input terminal 2. The storage section 3 includes a frame memory 31, a composite image memory 32, and a reference image memory 33. The frame memory 31 temporarily stores a current image needed for image combining. The composite image memory 32 temporarily stores a previous image and a composite image. The reference image memory 33 temporarily stores a reference image. The configuration terminal 4 sets a composition ratio α between the current and the previous images, where the composition ratio α is a weighting ratio of the current image to the composite image, and satisfies 0≦α≦1. The image combining section 5 combines the current and the previous images 8 and 9 at the composition ratio α set by the configuration terminal 4, and sequentially generates and outputs the composite image 10 to the composite image memory 32. The encoding section 6 receives image data 11 sequentially read from the composite image memory 32 in a format suitable for encoding moving pictures (e.g., in units of macro-blocks each formed of 16×16 pixels), encodes the image data 11, and outputs a data stream (compressed data). The output terminal 7 outputs the data stream (compressed data) encoded in the encoding section 6.
As shown in FIG. 14, the image combining section 5 includes a composition-ratio configuration section 51, multipliers 52 and 53, and an adder 54. The composition-ratio configuration section 51 outputs the composition ratio α set by the configuration terminal 4 to the multiplier 52, and a composition-ratio (1−α) to the multiplier 53. Note that the composition-ratio (1−α) is a weighting ratio of the previous image to the composite image. The multiplier 52 multiplies the values of current image 8 by the composition ratio α. The multiplier 53 multiplies the values of previous image 9 by the composition ratio (1−α). The adder 54 adds together the multiplication results of the multipliers 52 and 53, and generates the composite image 10. That is, a higher value of the composition ratio α causes the current image 8 to have alarger effect on the composite image 10, while a lower value of the composition ratio α causes the previous image 9 to have a larger effect on the composite image 10.
Next, a method to combine current and previous images, and then to sequentially encode the composite images using the image combining and encoding device 101 when images as shown in FIGS. 15A-15J are sequentially input will now be described with reference to the flowchart of FIG. 16.
First, the composition ratio α for the image combining section 5 is set from the configuration terminal 4, and the composite image memory 32 is initialized by writing values of zero to the entire area of the composite image memory 32 (ST1001).
<Process of First Frame (at Time T1)>
Then, a current image (input image) shown in FIG. 15A is input from the input terminal 2 at time T1, and is temporarily stored in the frame memory 31 (ST1002).
Next, the current image 8 temporarily stored in the frame memory 31 is sequentially read (ST1003). In parallel with this operation, image data 9 which are included in the previous image (at this stage, immediately after an initialization, all image data have values of zero) stored in the composite image memory 32 and which are located in a same region as that of the current image 8 in a two-dimensional space are sequentially read (ST1004). The images 8 and 9 are sequentially combined in the image combining section 5 (ST1005), and the composite image 10 is sequentially overwritten on the same region in the composite image memory 32; thus, the composite image 10 (the image shown in FIG. 15B) is temporarily stored in the composite image memory 32 (ST1006).
Next, the composite image 11 temporarily stored in the composite image memory 32 is read in a format suitable for encoding moving pictures (e.g., in units of macro-blocks each formed of 16×16 pixels), and is sequentially input to the encoding section 6 (ST1007). In the encoding section 6, intraframe predictive coding (I-picture coding) is sequentially performed on the composite image 11, and a data stream is output from the output terminal 7 (ST1008).
Next, the encoding section 6 performs local decoding such as a combination of dequantization and inverse DCT etc. to generate a reconstructed image, and temporarily stores the reconstructed image in the reference image memory 33 as the reference image 13 needed for interframe predictive coding of following frames (ST1009).
After the above steps ST1002-ST1009 are completed, at step ST1010, whether or not to terminate the encoding process is determined. If the process should continue to encode the composite image 11 of the next frame (No at ST1010), the process returns to step ST1002, and then a process similar to that described above is performed. If no more encoding is required (Yes at ST1010), all the process terminates.
<Process of Second Frame (at Time T2)>
Then, the current image shown in FIG. 15C is input from the input terminal 2 at time T2, and is temporarily stored in the frame memory 31 (ST1002).
Next, the current image 8 temporarily stored in the frame memory 31 is sequentially read (ST1003). In parallel with this operation, image data 9 which are included in the previous image (the image shown in FIG. 15B) stored in the composite image memory 32 and which are located in a same region as that of the current image 8 in a two-dimensional space are sequentially read (ST1004). The images 8 and 9 are sequentially combined in the image combining section 5 (ST1005), and the composite image 10 is sequentially overwritten on the same region in the composite image memory 32; thus, the composite image 10 (the image shown in FIG. 15D) is temporarily stored in the composite image memory 32 (ST1006).
Next, the composite image 11 temporarily stored in the composite image memory 32 is read in a format suitable for encoding moving pictures (e.g., in units of macro-blocks each formed of 16×16 pixels), and is sequentially input to the encoding section 6 (ST1007). In the encoding section 6, while the reference image 12 is sequentially read from the reference image memory 33, interframe predictive coding (P-picture coding) is sequentially performed on the composite image 11, and a data stream is output from the output terminal 7 (ST1008).
Next, the encoding section 6 performs local decoding such as a combination of dequantization and inverse DCT etc. to generate a reconstructed image, and temporarily stores the reconstructed image in the reference image memory 33 as the reference image 13 needed for interframe predictive coding of following frames (ST1009).
After the above steps ST1002-ST1009 are completed, at step ST1010, whether or not to terminate the encoding process is determined. If the process should be continued to encode the composite image 11 of the next frame (No at ST1010), the process returns to step ST1002, and then a process similar to that described above is performed. If no more encoding is required (Yes at ST1010), all the process terminates.
<Process of and after Third Frame (at Times T3-T5)>
The process of image combining and encoding of and after the third frame is similar to that for the second frame (at time T2) as described above. This process eventually causes the composite image 11 as shown in FIG. 15J to be encoded at time T5. The tracked trajectories of a moving object at respective times T1-T5 are drawn as indicated by the arrows; thus, a trajectory of a moving body can be visually perceived each time when recorded moving pictures are reproduced.
Technologies to combine images and to encode generated composite images are disclosed in Patent Document 1 and Patent Document 2.    Patent Document 1: Japanese Patent Publication No. 2002-044521    Patent Document 2: Japanese Patent Publication No. 2008-113112